reduction in test execution time
man hours saved in the first year
automated test checkpoints
Introduce an application test automation framework to support ambitious growth plans from a central IT service.
IT within TMNAS is delivered as a central service to all TMNAS group companies. When this was first introduced, Todd Soisson, Vice President of Business Experience for TMNA Services, had a very small Quality Assurance (QA) team that soon ran into capacity issues: “At the time we were using a freeware software testing tool for automated regression testing, and it just wasn’t cutting it. We were getting a new build every other day, and many new fixes were breaking older functions that had otherwise been working. Problem was, the freeware tool wasn’t catching everything and the scripts were difficult to maintain.”
Not long afterward, they also began receiving complaints from agents in the field regarding the slow performance of another core system. Further complicating the matter was the fact that test cases and associated metrics were being manually managed in spreadsheets. Soisson: “This testing model was obviously not scalable, and we saw our workload piling up, while releases were in danger of running late.” TMNAS realized test automation could be the answer and started investigating the market for solutions.
TMNAS offers the security of nearby expertise, enhanced by the diversity and power of one of the world’s most respected insurance groups. Tokio Marine’s companies offer access to leading commercial insurance solutions spanning the property and casualty landscape including professional liability, workers’ compensation, and property coverage.
Micro Focus UFT One is engineered to facilitate distributed test execution of multiple tests simultaneously. A proof-of-concept demonstrated that this could substantially speed up TMNAS’s testing cycle. Initially just one license was purchased, and Soisson was delighted with the results: “We went from three to four days of manual testing to an eight-hour automated overnight testing session. When we showed the CIO the ROI we were able to achieve with one license, he got really excited and challenged us to put a roadmap together to purchase more licenses, and start automating the entire regression test process. The early success with UFT One developed confidence in my team and quickly paved the way for additional Micro Focus solutions like ALM and LoadRunner Enterprise.”
Micro Focus ALM is an integrated IT software quality management solution that enables consistent QA procedures across the application lifecycle by enabling test case management and defect fixing in a centralized location. Micro Focus LoadRunner Enterprise is an automated performance and test automation solution that provides an accurate picture of end-to-end system performance under varying loads, so that issues can be identified and resolved before an application goes live.
TMNAS identified core test cases and built a framework to automate these. Over the next couple of years more and more test automation was added and a Quality Center of Excellence was established that included the newly formed automation team. Performance, functional, and end-to-end regression testing are all included and can be remotely executed and monitored through a Virtual Automation Lab, hosting 12 dedicated test automation VMs. Proactive dashboards and metrics provide a real-time view of test automation within TMNAS. Soisson: “We started with just one automation engineer and have grown the team to nine resources. We run the automation team as a development team rather than a test team, and we feel the value added to the business is what gives us our credibility.”
With 780,000 test checkpoints and 100 percent of regression tests being automated, all tier 1 applications undergo a monthly performance validation before being released into general production. This includes 150 hours of automated regression testing to cover all core applications, running high risk business workflows worth $4.5 billion to TMNAS.
The response from the business has been overwhelmingly positive: “If we have to make an emergency change to a key application, our internal clients feel safe in the knowledge that our test automation processes will run overnight and nothing catastrophic will have happened to their application when they come in in the morning. They trust our team and the processes we’ve introduced,” says Soisson.
Hemant Anugonda, Senior Manager of Quality Services at TMNAS, adds: “We’ve introduced parallel threading, leveraging TestNG, UFT Developer, IntelliJ, and Eclipse, running up to three testing threads on the same machine. This lowers test execution times significantly and enables us to release our versions much faster. The use of headless browsers (i.e. browsers without a graphical user interface) helps us run our test automation scripts without even rendering them on a screen. This allows us to reduce our regression test time by 70 percent, time we can use to execute deeper and broader application tests.”
TMNAS has ambitious plans for the future and the Quality Center of Excellence will play a central role in supporting them. It is moving towards a SaaS model to create more efficiencies and enable faster upgrades. In preparation for this, the foundations are being laid for a Test Data Management team to service test data requirements for TMNAS. They will enhance the automated testing framework as a self-service solution.
In response to increased use of mobile applications, TMNAS is expanding its mobile test capabilities. This triggers additional security requirements, and the Quality Center of Excellence is expanding its remit to include security testing as well. Integration of automated smoke tests with Octopus Deploy supports continuous integration and deployment, shortening the time to release new versions. A large infrastructure transformation program is underway, and the Quality Center of Excellence aims for all associated application testing to be automated and in a self-service model, through the use of Micro Focus solutions.
Soisson comments on the results:
Todd Soisson – VICE PRESIDENT OF BUSINESS EXPERIENCE
TMNA Services
ALM, he notes, has been especially helpful: “We were totally dependent on spreadsheets prior to buying ALM – it was a lot of manual work opening every spreadsheet to compile metrics across our enterprise. ALM took us to the next level – standard metrics are included, and you can design whatever else you can imagine.”
LoadRunner Enterprise has been so successful in helping the team test and ensure the performance of TMNAS core systems and other applications, that it’s now an integral part of the company’s standard operating procedure. Anugonda explains: “Micro Focus LoadRunner Enterprise makes it easy to benchmark the performance of your software, then rerun the same scripts after each deployment to make sure any deltas are acceptable – we’ve built an entire process around that and it’s now an integral part of our Quality Center of Excellence.”
The TMNAS team feels ready to support a move towards DevOps which is a roadmap item. Soisson:
Todd Soisson – VICE PRESIDENT OF BUSINESS EXPERIENCE
TMNA Services
He concludes: “We are very proud of the Quality Center of Excellence and the trust our employees have placed in us. The partnership with Micro Focus was key to its success. They listened, understood our long-term vision, and worked closely with us to help us achieve our goals as a business.”
